System and method for observing and controlling a programmable network using cross network learning

ABSTRACT

A system and method for observing and controlling a programmable network via higher layer attributes is disclosed. According to one embodiment, the system includes one or more collectors and a remote network manager. The one or more collectors are configured to receive network traffic data from a plurality of network elements in the network. The remote network manager is configured to connect to the one or more collectors over the Internet via a network interface. The one or more collectors extract metadata from the network traffic data and send the metadata to the network manager.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/827,571, filed Mar. 23, 2020, now issued as U.S.Pat. No. 11,374,812. U.S. patent application Ser. No. 16/827,571 is acontinuation application of U.S. patent application Ser. No. 16/584,810,filed Sep. 26, 2019, now issued as U.S. Pat. No. 10,630,547, and U.S.patent application Ser. No. 14/520,238, filed Oct. 21, 2014, now issuedas U.S. Pat. No. 10,601,654. U.S. patent application Ser. No. 16/584,810is a continuation application of U.S. patent application Ser. No.14/520,238. U.S. patent application Ser. No. 14/520,238 claims thebenefit of U.S. Provisional App. Ser. No. 61/893,789, filed on Oct. 21,2013. U.S. Provisional Patent Application 61/893,789, U.S. patentapplication Ser. No. 14/520,238, now issued as U.S. Pat. No. 10,601,654,U.S. patent application Ser. No. 16/584,810, now issued as U.S. Pat. No.10,630,547, and U.S. patent application Ser. No. 16/827,571 , now issuedas U.S. Pat. No. 11,374,812, are incorporated herein by reference.

FIELD OF THE DISCLOSURE

The present disclosure pertains generally to the fields of indexing andcontrolling networks. More particularly, the present disclosure relatesto a system and method for observing and controlling a programmablenetwork via higher layer attributes.

BACKGROUND

Obtaining business-level insight and control over the applications,users and devices in modern networks is becoming extremely challenging.On the applications front, modern networks have a huge mix inapplication types and deployment locations. For example, a singleapplication might be implemented as a distributed and multi-tierapplication with the inter-component communication running overdifferent parts of the network. Similarly, business applications may behosted off-premise in the cloud (e.g., salesforce.com), on-premise in alocal data center (e.g., SAP), or on-premise between hosts (e.g.,unified communications). On the users and devices front, modern networksare accessed by a myriad of devices from wired desktops to wirelessdevices such as laptop computers, mobile phones, and tablet PCs.

Traditional network security and performance monitoring tools or policyenforcing firewalls require dedicated hardware deployed inline with userdevices. However, dedicated hardware has drawbacks in supporting varioustypes of applications and devices deployed in different parts of thenetwork.

SUMMARY

One embodiment of the present disclosure is a system for monitoring anetwork. In this embodiment, the system includes one or more collectorsand a remote network manager. The one or more collectors are configuredto receive network traffic data from a plurality of network elements inthe network. The remote network manager is configured to connect to theone or more collectors over the Internet via a network interface. Theone or more collectors extract metadata from the network traffic dataand send the metadata to the network manager.

According to some embodiments, the system further has a programmablecontroller that controls at least some of the plurality of networkelements. The remote network manager controls the plurality of networkelements via the programmable controller. The remote network manager orthe one or more collectors are further configured to index the networkenabling efficient search and retrieval of the metadata.

According to some embodiments, the collector receives mirror trafficdata from the plurality of network elements and indexes the network. Themanager programs the programmable network element to send filterednetwork traffic data from the plurality of network elements to thecollector. The filtered network traffic data is used for networkanalysis and extracting the metadata. The collector is furtherconfigured to receive statistics about the network, topology informationabout the network, input from one or more enterprise systems, orcombinations thereof.

According to some embodiments, the collector collects the filterednetwork traffic data based on a time-varying schedule. The time-varyingschedule for sending the filtered network traffic data is determined bya bandwidth constraint at the collector, and/or a network topology and anetwork policy.

According to some embodiments, the manager time-aligns the metadatareceived from the network traffic data with data received from theenterprise system. The manager applies a control loop to determinewhether a high level control objective is met after programming theprogrammable network element. The manager is further configured tosimultaneously and centrally analyze a network condition of a pluralityof networks, learn a pattern from a first network of the plurality ofnetworks, and apply the pattern to a second network of the plurality ofnetworks. The manager extracts a lower layer control primitive affectinga network policy and programming the programmable network element basedon the lower layer control primitive such as an access control list(ACL), quality of service (QoS), rate limit settings, or combinationsthereof. The manager maintains a relationship between a high levelnetwork policy and the low level control primitive.

Another embodiment of the present disclosure is also a system formonitoring a network. This embodiment includes a collector that isconfigured to communicate with a programmable switch that receivesnetwork traffic from a plurality of network elements of the network. Inthis embodiment, the collector is configured to receive the networktraffic, extract features from the network traffic, and program theprogrammable switch to receive filtered network traffic from one or moreof the plurality of network elements.

Yet another embodiment of the present disclosure is also a system formonitoring a network. This embodiment also includes a collector that isconfigured to communicate with a programmable switch that receivesnetwork traffic from a plurality of network elements of the network. Thecollector is further configured to receive the network traffic, extractfeatures from the network traffic, and index the network based on highlayer information, wherein the high layer information is one or more ofnetwork users, network applications, network devices, and networkbehaviors. The collector is further configured to program theprogrammable switch to receive filtered network traffic from one or moreof the plurality of network elements.

Yet another embodiment of the present disclosure is a system formonitoring a network, where the system includes a collector and amanager. In this embodiment, the collector is configured to communicatewith a programmable switch that receives network traffic from aplurality of network elements of the network, and the collector isconfigured to receive the network traffic, extract features from thenetwork traffic, and program the programmable switch to receive filterednetwork traffic from one or more of the plurality of network elements.The manager is configured to communicate with the collector to receivethe extracted features from the collector, summarize the extractedfeatures, and index the network based on high layer information, whereinthe high layer information is one or more of network users, networkapplications, network devices, and network behaviors. The manager can,for instance, be located in the cloud. This embodiment can include aplurality of collectors.

Yet another embodiment of the present disclosure is a system formonitoring a network that includes a collector and a manager. In thisembodiment, the collector receives network traffic from a plurality ofnetwork elements of the network, and extract features from the networktraffic. The manager is configured to communicate with the collector toreceive the extracted features, summarize the extracted features, andindex the network based on high layer information, wherein the highlayer information is one or more of network users, network applications,network devices, and network behaviors. In addition, this embodiment caninclude a plurality of collectors.

Yet another embodiment of the present disclosure is a system formonitoring a network, where the system includes a programmable switchand a collector. The programmable switch is configured to receivenetwork traffic from a plurality of network elements of the network. Thecollector is configured to communicate with the programmable switch, andthe collector is further configured to receive the network traffic,extract features from the network traffic, and program the programmableswitch to receive filtered network traffic from one or more of theplurality of network elements. In this embodiment, the collector isfurther configured to receive statistics about the network, topologyinformation about the network, input from other enterprise systems, orcombinations thereof. In addition, the collector is further configuredto index the network enabling efficient search and retrieval of themetadata.

Another embodiment of the present disclosure is a system for controllinga network. This embodiment includes a collector that is configured tocommunicate with a programmable switch that receives network trafficfrom a plurality of network elements of the network, wherein thecollector is configured to receive the network traffic, extract featuresfrom the network traffic, and program one or more of the networkelements to enforce one or more policies.

Another embodiment of the present disclosure is a system for controllinga network, where the system includes a programmable switch and acollector. In this embodiment, the system includes a programmable switchthat is configured to receive network traffic from a plurality ofnetwork elements of the network, and a collector that is configured tocommunicate with the programmable switch, wherein the collector isfurther configured to receive the network traffic, extract features fromthe network traffic, and program one or more of the network elements toenforce one or more policies. At least one of the one or more policiescan be based on security or performance issues with the network.

Yet another embodiment of the present disclosure includes a system formonitoring and controlling a network. This embodiment includes acollector that is configured to communicate with a programmable switchthat receives network traffic from a plurality of network elements ofthe network. In addition, in this embodiment, the collector isconfigured to receive the network traffic, extract features from thenetwork traffic, program the programmable switch to receive filterednetwork traffic from one or more of the plurality of network elements,and program one or more of the network elements to enforce one or morepolicies.

In one embodiment, the present system also referred to herein as theLoupe System, crawls, summarizes, indexes, queries, and/or controlsnetworks. The networks can include a combination of physical and virtualnetwork elements. Some embodiments of the present disclosure providehigher layer awareness and instrumentation of such networks where theunderlying network elements may or may not have that higher-layerprocessing capability.

In one embodiment, the present disclosure can relate to visibilityissues, such as crawling, summarizing, indexing and querying. In thisembodiment, the visibility part of the method and system entailsextracting key features from different parts of the network and bindingthese features to higher layer information such as users, applications,devices and behaviors. This higher layer information can then be storedand made query-able via natural language processing, and a ranking ofresponses can be computed and presented to the user.

Another embodiment of the present disclosure relates to controlling anetwork. According to one such embodiment, the control part entailsusing the information from the visibility part to enforce high-levelpolicies and automatically remediating security and performance issuesin the network. In some embodiments, one technique is to, dynamicallyand in real-time, track the binding of higher layer information to thespecific lower layer primitives that the physical and virtual networkelements understand and can be programmed with. This embodimentdynamically programs the devices via the lower layer primitives, therebyachieving the desired higher layer objective.

The disclosed embodiments further relate to machine readable media onwhich are stored embodiments of the disclosed invention described inherein. It is contemplated that any media suitable for retrievinginstructions is within the scope of the disclosed embodiments. By way ofexample, such media may take the form of magnetic, optical, orsemiconductor media. The disclosed embodiments also relate to datastructures that contain embodiments of the disclosed invention, and tothe transmission of data structures containing embodiments of thedisclosed invention.

Further aspects of the disclosed embodiments will be brought out in thefollowing portions of the specification, wherein the detaileddescription is for the purpose of fully disclosing the variousembodiments without placing limitations thereon.

BRIEF DESCRIPTION OF THE DRAWING

The present application will be more fully understood by reference tothe following figures, which are for illustrative purposes only. Thefigures are not necessarily drawn to scale and elements of similarstructures or functions are generally represented by like referencenumerals for illustrative purposes throughout the figures. The figuresare only intended to facilitate the description of the variousembodiments described herein. The figures do not describe every aspectof the teachings disclosed herein and do not limit the scope of theclaims.

FIG. 1 illustrates a functional diagram of a SDN-enabled network,according to one embodiment;

FIG. 2A illustrates system architecture of an exemplary system deployedin an enterprise network, according to one embodiment;

FIG. 2B illustrates system architecture of an exemplary system deployedin an enterprise network, according to another embodiment;

FIG. 3 is a block diagram of an out-of-band deployment, according to oneembodiment;

FIG. 4 is a block diagram of an inline deployment, according to oneembodiment;

FIG. 5 is a flow diagram for providing network visibility, according toone embodiment;

FIG. 6 is a flow diagram of an input collection process at thecollector, according to one embodiment;

FIG. 7 illustrates a diagram of an exemplary SDN enabled network,according to one embodiment;

FIG. 8 illustrates a diagram of an exemplary of legacy network includinga SDN-enabled switch, according to one embodiment;

FIG. 9 is a flow diagram of an exemplary information collection process,according to one embodiment;

FIG. 10 is a flow diagram of summarization and indexing processes,according to one embodiment; and

FIG. 11 is a flow diagram of a control loop, according to oneembodiment.

FIG. 12 illustrates an exemplary computer architecture, according to oneembodiment

DETAILED DESCRIPTION

Persons of ordinary skill in the art will understand that the presentdisclosure is illustrative only and not in any way limiting. Otherembodiments of the presently disclosed system and method readily suggestthemselves to such skilled persons having the assistance of thisdisclosure.

Each of the features and teachings disclosed herein can be utilizedseparately or in conjunction with other features and teachings toprovide a system and method for observing and controlling a programmablenetwork via higher layer attributes. Representative examples utilizingmany of these additional features and teachings, both separately and incombination, are described in further detail with reference to theattached figures. This detailed description is merely intended to teacha person of skill in the art further details for practicing aspects ofthe present teachings and is not intended to limit the scope of theclaims. Therefore, combinations of features disclosed above in thedetailed description may not be necessary to practice the teachings inthe broadest sense, and are instead taught merely to describeparticularly representative examples of the present teachings.

In the description below, for purposes of explanation only, specificnomenclature is set forth to provide a thorough understanding of thepresent system and method. However, it will be apparent to one skilledin the art that these specific details are not required to practice theteachings of the present system and method.

Some portions of the detailed descriptions herein are presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the below discussion, itis appreciated that throughout the description, discussions utilizingterms such as “processing,” “computing,” “calculating,” “determining,”“displaying,” “configuring,” or the like, refer to the actions andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present application also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but not limited to, any type of disk,including floppy disks, optical disks, CD-ROMs, and magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus.

The algorithms presented herein are not inherently related to anyparticular computer or other apparatus. Various general purpose systems,computer servers, or personal computers may be used with programs inaccordance with the teachings herein, or it may prove convenient toconstruct a more specialized apparatus to perform the required methodsteps. The required structure for a variety of these systems will appearfrom the description below. It will be appreciated that a variety ofprogramming languages may be used to implement the teachings of thedisclosure as described herein.

Moreover, the various features of the representative examples and thedependent claims may be combined in ways that are not specifically andexplicitly enumerated in order to provide additional useful embodimentsof the present teachings. It is also expressly noted that all valueranges or indications of groups of entities disclose every possibleintermediate value or intermediate entity for the purpose of originaldisclosure, as well as for the purpose of restricting the claimedsubject matter. It is also expressly noted that the dimensions and theshapes of the components shown in the figures are designed to help tounderstand how the present teachings are practiced, but not intended tolimit the dimensions and the shapes shown in the examples.

There are four main areas of technology that may incorporate embodimentsof the present disclosure: (1) network functions virtualization (NFV),(2) software defined networking (SDN), (3) application delivery network(AND), and (4) network packet brokers and network security andperformance monitoring tools. The present system and method bringstogether aspects of these technologies to provide visibility as well ascontrol of networks.

FIG. 1 illustrates a functional diagram of a SDN-enabled network,according to one embodiment. The SDN-enabled network 100 includes SDNapplications 110 and network elements 120 that are linked via one ormore SDN controllers 115A-115N. The “p” and “v” prefixes on the networkelements 120 indicate physical and virtual network elements,respectively.

The network elements 120 include a physical switch (pSwitch) 121, aphysical router (pRouter) 122, a physical Firewall (pFirewall), avirtual switch (vSwitch) 124, a virtual firewall (vFirewall) 125, and aphysical network packet broker 126. It is appreciated that the networkelements 120 can include any number of physical switches 121, physicalrouters 122, physical firewalls 123, virtual switches 124, virtualfirewalls 125, and physical network packet broker 126, and otherphysical or virtual network elements, without deviating from the presentdisclosure.

Network functions virtualization (NFV) refers to the implementation anddeployment of software-based network elements. Such software-basednetwork elements typically run on generic processing hardware (e.g., x86machines) as opposed to non-NFV network elements that require dedicatedhardware (e.g., Application-Specific Integrated Circuits (ASICs)).Examples of NFV-type network elements include, but are not limited to, avirtual switch 124 and a virtual firewall 125. It is appreciated thatother types of NFV-type network elements may be implemented withoutdeviating from the present disclosure. Such NFV-type network elementsmay be run as a virtual machine on top of a hypervisor that runs oncommodity hardware. The present system and method provides monitoringand controlling of NFV network elements, but it is noted that thepresent system and method can also monitor and control non-virtualizednetwork elements and/or functions without deviating from the presentdisclosure.

Software defined networking (SDN) describes the generic concept ofseparating the entirety or some portion of the control plane from thedata plane of network elements. For simplicity, the term “networkelement” herein can refer to a physical, a virtual network element, or acombination of both.

The separate portion of the control plane is typically centralized in aSDN controller. The southbound interfaces 152 between the SDN controller115 and the network elements 120 can be open (e.g., OpenFlow®) orproprietary (e.g., onePK®). The SDN controller 115 provides programmaticnorthbound interfaces 151 for SDN applications 110 to both observe anddynamically configure network elements. Similar to a SDN application,the present system and method utilizes the northbound interfaces 151between the SDN applications 110 and the SDN controller 115. It is notedthat the present system and method can work with a non SDN-enablednetwork, a partially or fully enabled SDN network, or even a networkincluding heterogeneous networks. For example, the SDN controller 115 ofFIG. 1 and a wireless controller 259 of FIG. 2 are an example of anenterprise system.

An application delivery network (ADN) encapsulates several technologiesthat provide application-layer functionality in the network. A nextgeneration application firewall, for example, is an appliance thatprovides inline access control functionality as a function of L4-L7header information as well as application, user and content layermetadata. This appliance can perform inline deep packet inspection toidentify in real-time applications and perform access control.

The control embodiments of the present system and method providescapabilities of the next generation application firewall using basicnetwork elements such as switches and routers that otherwise would nothave such capability. The present system and method can reduce hardwareand distributed functionality.

The network packet broker 126 (or a matrix switch) gathers, aggregatesand filters network traffic from port mirrors, network TAPs, and probes.The network packet broker 126 serves the filtered network traffic tonetwork security and performance tools as per their network security andperformance tools. For example, a network security and performance toolmay only support 1 GBps of traffic, and a network packet broker 126 canbe manually configured to filter and shape traffic from a 10 GBps linkto conform to the constraint of the network security and performancetool. Typically the network packet broker 126 is decoupled from thenetwork security and performance tools to which it delivers the packets.

A portion of the present system and method performs as a networksecurity and performance tool. In one embodiment, the present system andmethod intelligently and dynamically programs a network packet broker126 to gain access to the traffic it needs. The present system andmethod also summarizes and indexes higher layer information about users,applications, devices, behaviors, and the like (e.g., via machinelearning), and enables the higher layer information to be queried usinga natural language processing technique. According to one embodiment,the present system and method is deployed in a cloud to enablecross-network learning. “Cloud” herein refers to a computer and storageplatform service hosted over a wide area network (e.g., the Internet).It is noted that both ADN and network security/performance monitoringtools are typically deployed on premise.

The present system and method observes and controls a programmablenetwork via higher layer attributes and addresses the drawbacks of priorsystems for monitoring and controlling networks. The discussion isdivided into three sections: (1) architecture, (2) visibility, and (3)control.

Architecture

FIG. 2A illustrates system architecture of an exemplary system deployedin an enterprise network, according to one embodiment. The system 200includes a manager 201 (herein also referred to as Loupe Manager) andone or more collectors 202 (herein referred to as Loupe Collectors). Inone embodiment, the collector 202 is a software appliance (virtual orphysical) that is located on premise. The collector 202 may be deployedas a single software element, or for scaling a cluster or severalsoftware elements. For example, the collector 202 is logic in anon-transitory computer readable memory that can be executed by aprocessor to perform the actions described herein. In other embodiments,the collector 202 is a combination of hardware and software.

According to some embodiments, there are multiple collectors 202 perenterprise network 210 (e.g., a campus, a data center) and multiplenetworks 210 and collectors 202 per customer. Moreover, the collectors202 can be deployed behind firewalls within an enterprise network 210.This enables the collectors to easily communicate with enterprisesystems on-premise and also behind the firewall to easily communicateoutbound with systems off-premise.

The collector 202 receives live packets captured directly from physicaland/or virtual network elements 216. The collector 202 also receivesdata (e.g., topology, statistics, user information, and the like) fromother enterprise systems including identity management systems (e.g.,active directory 217), network element controllers (e.g., SDNcontrollers 215, network management systems), and the like. Thecollector 202 also runs performance tests against on/off-premiseapplications in the public cloud/Internet 250 (e.g., BOX®, MICROSOFTOFFICE365®, GOOGLE®, WEBEX®, WORKDAY®, SALESFORCE®) and collects theperformance results.

The collector 202 captures all of these data, extracts key metadata orfeatures, and compresses and sends the key metadata or features to themanager 201 that is located in a public cloud 220. For example, thecollector 202 receives 10s or 100s of gigabits per second of data, butonly sends 10s or 100s of kilobits per second of data to the manager201. The collector 202 is provisioned and configured by the manager 201,thus the commands from the manager 201 towards systems that areon-premise can be proxied via the collector 201. In one embodiment, themanager 201 may also be deployed in a private cloud or elsewhere withina large multi-site organization.

The manager 201 summarizes and stores the data received from thecollector 202 in a database 205. The manager 201 performs additionaldata collection from off-premise enterprise systems and otherapplications over the public cloud/Internet 250 and runs its ownperformance test. The manager 201 applies learning and other heuristicalgorithms on the data and bind higher-layer information (e.g., aboutusers, applications, devices, and behaviors) to the data. The manager201 also computes the crawling schedule for the collectors 202 toreceive data from different parts of the network. The manager 201 isalso responsible for providing a web interface and a natural languagequery capability to retrieve ranked answers based on the learned data.Similar to the collector 202, the manager 201 is a software appliancethat can be deployed in a cluster or in multiple tiers. The manager 201contains a database 205 that can support large data storage andefficient queries (e.g., BigTable®). Generally, there can be one manager201 for many organizations and/or enterprises (e.g., multi-tenant styledeployment), or multiple managers 201 for multiple organizations and/orenterprises. The manager 201 may also be logic in a non-transitorycomputer readable memory that can be executed by a processor to performthe actions described herein or a combination of hardware and software.

FIG. 2B illustrates system architecture of an exemplary system deployedin an enterprise network, according to one embodiment. The system 250includes a manager 251, a collector 252, a wireless controller 265 thatcontrols one or more wireless access points (APs) 256. The wirelesscontroller 265 may take many forms, for example, (i) a separateon-premise software running on its own hardware, (ii) software that isintegrated into the access points 256, or (iii) software locatedoff-premise (e.g., in a cloud 220). The wireless controller 265 controlsand/or configures the access points 256 and terminates data planetraffic coming from mobile devices that are wirelessly connected to theaccess points 256. The wireless controller 265 is an example of anetwork element, as well as a SDN controller that controls several othernetwork elements (e.g., access points 256).

The collector 252 collects wireless metrics from the controller 265 viaa management interface (e.g., simple network management protocol (SNMP),command-line interface (CLI), proprietary management protocol). Examplesof these metrics for a mobile device include, but are not limited to:signal strengths, layer 2 traffic statistics (e.g., packets transmitted,retried, dropped), traffic transmission rates, device location, and userinformation. Examples of these metrics for an access point include, butare not limited to: channel utilization, aggregated layer 2 trafficstatistics, interference measurements, CPU/memory utilization.

The collector 252 simultaneously collects metrics and other informationfrom other enterprise systems where available, via their respectivemanagement interfaces. One example is collecting user role as well asuser-to-IP address information from a directory server (e.g., LDAP,Active Directory). Another example is collecting unified communicationperformance metrics from a Microsoft Lync Server).

The collector 252 simultaneously sees network traffic via a mirroredinterface via a logical or physical port mirror off of the wirelesscontroller 265, or a logical or physical port mirror off of anothernetwork element (e.g., switch, router, access point) in the networkwhere relevant user traffic is carried.

From the traffic, the collector 252 performs deep packet inspection(DPI) and extracts, in addition to general protocol level metadata,user/device quality of experience (QoE) related metadata, differing onan application-by-application basis. For example, web browsing QoEmetrics include page load times and/or HTTP URL response times. Voiceand video application QoE metrics involve extracting and/or computingthe relevant mean opinion score (MOS) values.

According to some embodiments, the present system and method time alignsthe QoE metadata with metadata extracted across the application stackincluding the wireless layer metrics from the wireless controller 265.For example at a particular time interval, a user/device may have poorpage load times, high transmission control protocol (TCP) retransmits,low signal-to-noise ratio (SNR), high AP channel utilization. Thepresent system and method collects and stores this time series data, andanalyzes the time series data for trends/patterns over time and otherdimensions (e.g., device type, location). For example, the presentsystem and method finds that ANDROID® devices suffer consistently worseweb performance than IOS® devices.

According to some embodiments, the present system and method analyzesfor trends/patterns is across networks. For example, the present systemand method identifies the specific network/protocol/wireless metrics todetermine the application performance. As an example, the present systemand method analyzes a bad Microsoft Lync® voice application performance(e.g., mean opinion score (MOS)) across many customer networks. Thepresent system and method learns that the most important indicator ishigh levels of layer 2 packet retransmissions. Based on this assessment,the present system and method predicts, for a new customer network thathas high levels of layer 2 packet retransmissions, that Microsoft Lync®performance would be poor unless the packet retransmissions problem isrectified.

The present system and method has applicability to two use cases:visibility and control. From an architecture perspective, there is adifference between deployment possibilities between the two use cases.In particular, for passive visibility only, the present system andmethod can be deployed out-of-band. FIG. 3 is a block diagram of anout-of-band deployment, according to one embodiment. A programmable(e.g., SDN-enabled) switch 324 receives mirrored traffic of networkelements 316 and communicates with a server 350 including a collector302 and a controller 330. The mirrored traffic (as indicated in dottedlines) from the network elements 316 is forced through the programmableswitch 324. The programmable switch 324 can be dynamically controlledand programmed to direct specific traffic during specific time intervalsand network locations to the collector 302. For example, the controller330 controls the programmable switches 324. In a case where the totalbandwidth of the traffic being monitored is less than the bandwidth ofthe collector 302, the programmable switch 324 may not be necessary andall mirrored traffic can be directly sent to the collector 302. Anexample of this case is where only the wide area network (WAN) linkswithin an enterprise network are monitored.

For control, the present system and method employs an inline deployment,according to some embodiments. In this case, a subset of the networkelements carrying regular traffic (e.g., non-mirrored traffic) isprogrammable (e.g., SDN-enabled). Moreover, these network elements(e.g., physical and virtual switches, wireless access points) may belocated such that the policy can be effective, for example, to form aphysical or logical choke point. FIG. 4 is a block diagram of an inlinedeployment, according to one embodiment. A manager 401 receives trafficfrom non programmable network elements 416 and programmable networkelement 417 and communicates with a server 450 including a collector 402and a controller 430. In this embodiment, the manager 401 is deployed onpremise in a private cloud 410 but it is apparent that the manager 401can be deployed off-premise in a public cloud as illustrated in FIGS. 2Aand 2B.

The manager 401 located in a cloud is capable of observing acrossmultiple customer networks. While the manager 401 (whether it is amulti-tenant manager or a separate manager per customer) may be deployedin a private or public cloud to preclude sharing of data across multiplenetworks, the present system and method may achieve overall performanceimprovement by combining trained algorithms from each of the customernetworks.

Visibility

The present system and method provides crawling and indexing the networkand enables natural language query about the network and applications,users, devices and behaviors. The specific flow for network visibilityis in the following order:

RAW DATA→CRAWLING→FEATURESEXTRACTION→SUMMARIZATION→INDEXING→CROSS-NETWORK-LEARNING→QUERY-ABILITY

FIG. 5 is a flow diagram for providing network visibility, according toone embodiment. The RAW DATA→CRAWLING→FEATURES EXTRACTION pieces occuron premise (e.g., collectors of Company 1 (510A), a branch of Company 1(501B), Company 2 (502)), and the SUMMARIZATION→INDEXING→CROSS NETWORKLEARNING→QUERY-ABILITY pieces occur in a cloud 550 (e.g., managers ofCompany 1 (511) and Company 2 (512)). It is noted that thesefunctionalities may be split across a manager and a collector in avariety of ways without deviating from the scope of the presentdisclosure. For example, partial summarization can occur in thecollector as opposed to the manager.

Raw Data

The raw data includes data that can be collected or crawled by acollector or a manager. The first piece of raw data that is crawled is alive traffic on the network that can be accessed by one or morecollectors. The raw data can further include statistical, topologicaland configuration data—received either from network elements directly,or via an intervening controller or a manager. Examples of raw datainclude, but are not limited to, sampled flow (sFlow®) and SNMP dataharvested from network elements. Similarly, topology information can begleaned from a SDN controller if available. Other information gleanedfrom other enterprise systems (on or off-premise) is also applicable,for example, user information received from an ACTIVE DIRECTORY® server.

The raw data also includes the results from pro-active performance testswith respect to on and off-premise applications. In one embodiment, thecollector runs proactive performance tests (e.g., HTTP GETs, PINGs) withvarious target applications. These target applications can beautomatically detected by the present system and method or specificallyuser pre-configured.

Crawling Raw Data

Crawling herein refers to an act of dynamically selecting a differentset of raw data for the collectors to examine at any given time. Forexample, crawling includes observing different physical or virtuallinks, and applying different filters to the raw data.

In many cases, the total amount of traffic exceeds the bandwidth of acollector. This necessitates a device with network packet brokerequivalent (NPBE) functionality that is capable of driving mirrored andfiltered traffic from multiple parts of the network to the collector.The present system and method dynamically programs one or more NPBEdevices with filtering and steering rules to get selected access to thedata. However, the present system and method also is applicable to acase where the traffic mirrored to the collector comes from a smallnumber of locations (e.g., mirrored traffic from WAN links), and whenthe total simultaneous mirrored traffic is less than the bandwidth ofthe collector. This case may not require a NPBE device. In oneembodiment, the NPBE is one or more software elements, for example,running as part of the collector.

Crawling the raw data is a significant problem especially in situationswhere the present system and method can dynamically control one or moreNPBEs within the network to capture packets from different parts of thenetwork at different times. In one embodiment, a NPBE functionality isimplemented by a SDN controller operating on top of a SDN-enabledswitch. In this case, the manager, either directly or proxied via thecollector, can command the SDN controller to have the underlying networkelements to implement the NPBE functionality.

The method for controlling the network packet broker equivalent is forthe manager to compute a dynamic crawling and filtering schedule thatinforms the NPBE on how it may steer traffic to the collector. Thecomputation of the dynamic crawling and filtering schedule may be donein a variety of ways, for example, but not limited to, as a function oftopology, computation and network resources at the collector, andstatistics.

An example of a dynamic crawling and filtering schedule is:

-   -   Send all ingress and egress traffic from link e1 to the        collector;    -   From link e2, send ingress and egress traffic with source or        destination port equal to 80 to the collector; and    -   Cycle through links e3, e4, e5 and e6, 5 minutes at a time,        sending all traffic to the collector.

A dynamic crawling and filtering schedule with more complicated logicmay be sent to the collectors. For example, collectors can beprovisioned with a program that searches for a dynamic trigger to alterthe schedule. For example, the dynamic trigger is: “if an application Xis detected and is using Y bandwidth, then monitor traffic from the linkmore frequently.” In another embodiment, the dynamic crawling andfiltering schedule is computed to optimize load balancing betweencollectors, for example, “send the 1 GBps of traffic from link e1 tocollector #1 and the 1 GBps of traffic from link e2 to collector #2.”

According to one embodiment, the collector crawls performanceinformation of on- and off-premise applications that the present systemand method detects use of, or is pre-configured by a user. Theperformance information may be generated by the collector performingperformance tests (e.g., PING, TRACEROUTE, HTTP GETs) against theapplications. The performance information can be crawled by periodicallyrunning the same HTTP GETs against a target application that ispre-configured or automatically detected, and sending to the manager thedetected results. The crawling schedule may include a command, forexample, “if a new application is detected, then immediately startrunning performance tests against it.”

According to some embodiments, the raw data can be collected from a SDNcontroller or a network management system in the following process:

-   -   Global view of L1→L7 Network Topology,    -   Port statistics for each network element, if available,    -   Current Configuration of each network element under control,    -   Configuration Capability of each network element under control,    -   API functionality and configuration capabilities of the        controller itself,    -   Any higher layer information available regarding users,        applications, devices, locations, and the like.

According to some embodiments, the raw data can be collected from anenterprise system (e.g., ACTIVE DIRECTORY®, light directory accessprotocol (LDAP) servers, single sign-on (SSO) system). Examples of suchraw data include, but are not limited to, user information such as rolesand associated policies, login status, IP address.

According to some embodiments, the raw data can be collected fromnetwork elements directly (e.g., by way of a priori instructions givento a SDN controller) in the following process:

-   -   Sampled mirrored traffic from various ports in the network,    -   Advanced statistics such as sFlow®, netFlow®,    -   Previously computed information regarding users, applications,        devices, locations, and    -   Signal strength, error-rate, and other performance related        information.

According to some embodiments, the raw data can be collected from thepresent system or other policy engine such as desired high levelpolicies. According to some embodiments Performance data generated bythe collector including results of proactive tests (e.g., PING, HTTP,TCP) performed by the collector on detected or user pre-configuredon/off premise applications.

FIG. 6 is a flow diagram of an input collection process at thecollector, according to one embodiment. The input collection processstarts (at 610) and a collector receives inputs from a manager (at 602).Examples of inputs include, but are not limited to:

-   -   instructions on which enterprise systems to collect data from        and how to collect the data (e.g., IP address, credentials),    -   sampling schedule for data collection from network elements,    -   instructions on initial analysis, filtering, and compression of        collected data, and    -   list of applications to run performance test.

The collector further sends desired tapping configuration to the SDNcontroller and receives network topology (at 603), contacts theenterprise system and requests a stream of data to analyze (at 604),receives sampled raw data streams identified by time and link (at 605)and extracts features from the received sampled raw data streams perinstructions (at 606), receives advanced statistics from networkelements (at 607), and performs application performance tests andcollects data (at 608). The controller further extracts features usinginformation collected from 603-608 and compresses collected information(at 609). The controller sends data to the manager (at 610), and repeatsthe input collection process.

Feature Extraction

According to one embodiment, the present system and method extracts keyfeatures and/or metadata from the crawled data. For example, packets arestreaming into the collector at multiple gigabits per second speeds. Thecollector extracts a set of features on a flow-by-flow, or ahost-by-host basis from millions of packets per seconds and tens ofthousands of flows per second, and sends the extracted data to themanager in less than a few hundred bytes per second per flow. In oneembodiment, a flow is defined by the 5-tuple of (src1P, dst1P, srcPort,dstPort, protocol). The definition of a flow may be expanded to apply toother primitives such as application or other combinations of packetheader fields (e.g., Layer 2 flows include source and destination mediaaccess control (MAC) addresses in the definition of a flow).

Examples of a flow-by-flow feature include, but are not limited to:

-   -   Number of different HTTP2xx RESPONSE packets    -   Number of different HTTP3xx RESPONSE packets    -   Number of different HTTP5xx RESPONSE packets Binary feature of        whether IP Traffic is present    -   Number of different types of HTTP packets    -   Number of different types of DNS packets    -   Number of different types of DHCP packets    -   Binary feature of whether TCP_SYN was followed by TCP_SYN ACK    -   Binary feature of whether DNS_Q was followed by DNS_SUCC_RESP    -   Binary feature of whether DHCP_REQUEST was followed by        DHCP_GRANT    -   Set of source/destination MAC addresses present in the flow    -   Each of the above features on a time slice by time slice basis        (e.g., every 10 seconds of a flow)    -   Mean, median and variance of packet inter-arrival times, payload        sizes    -   Flag indicating whether window scaling was requested    -   Number of TCP FIN packets seen

Examples of a host-by-host feature include, but are not limited to:

-   -   Number of different hosts a particular host interacts with    -   Set of hosts that interact with each other    -   Number of ports used for transactions (indicates server vs.        client)

Examples of application-level metadata include, but are not limited to:

-   -   HTTP response and page load times    -   Voice and video call MOS scores    -   Response times of other protocols (DNS, DHCP, RADIUS, etc.)

Small raw data (e.g., statistics, topology) can be compressed and sentto the manager. However, intelligent feature extraction is required tosend a large data to the manager. An example of a large data isstatistical data (e.g., average link utilization). Similarly, theperformance test results might be reduced down to specific features(e.g., average HTTP response time, presence of an anomaly in theperformance test).

EXAMPLES

FIG. 7 illustrates a diagram of an exemplary SDN enabled network,according to one embodiment. Seven switches s0-s6 and network elementsh0-h2 are arranged hierarchically. The top switch s0 is connected to theInternet 750, and a manager 701 is deployed in a server in the publiccloud and connected via the Internet 750. A collector 702 is deployed asa virtual machine (VM) on a server attached to switch s6. The switchess0-s6 are SDN enabled switches and a SDN controller 715 is deployed as aserver attached to switch s5. An active directory server 725 is alsoconnected to switch s5.

FIG. 8 illustrates a diagram of an exemplary of legacy network includinga SDN-enabled switch, according to one embodiment. Seven switches s0-s6feed mirrored traffic (as indicated by dotted lines) into a SDN-enabledswitch 824. The mirroring configuration is static, and as an example maysimply mirror the traffic from each switch's uplink. The collector 802and SDN controller 815 are deployed connected to ports connected to theSDN-enabled switch 824. The manager 801 is deployed in a server in thepublic cloud and connected to the switch s0 over the Internet 850. Anactive directory server 825 is also connected to switch s5. It is notedthat mirror ports can be manually configured without the presence of aSDN enabled switch.

The collector 802 dynamically captures packets from multiple links inthe network. As an example, the link to the collector is a 2 GBps link(e.g., 2 link-aggregated IGBps links), and other links (including theWAN link) are IGBps links. In this case, the manager may send a crawlschedule to the collector, for example:

-   -   Collect the features on the WAN link (e0) 100% of the time, and    -   Continuously cycle through links e3, e4, e5, e6 (i.e., certain        of the depicted links) for five minute stretches, and collect        all the features during that time.

Summarization and Indexing

Summarization and indexing functionalities are implemented in a manageralthough it is possible to embed some or all of this functionality in acollector as well. The summarization and indexing processes take inputfeatures and other relevant data from the collector(s) and othersystems. The first outputs of the summarization and indexing processesare higher layer inferences, or bindings. Specifically, the relationshipor binding of higher layer data (e.g., users, applications, devices) tolower layer data (e.g., IP and MAC addresses, ports) is computed andindexed in a database. The present system and method provides acapability to query using natural language and high-layer controlprimitives, and any high-level indexed information, both current andhistorical.

The lower layer data may vary depending on an objective such as networkvisibility or network control. For network visibility, the lower layerdata includes, but is not limited to, protocol level metrics andmetadata. For network control, the lower layer data includes, but is notlimited to, control primitives such as ports, MAC addresses, IPaddresses, an access control list (ACL), quality of service (QoS), andrate limit setting. According to one embodiment, the present system andmethod predicts performance of one or more of an application, a user,and a device based on observed characteristics of the network aroundnetwork protocol level metrics and metadata.

The main role of the summarization process is to store and learn fromthe inputs received from the collector(s) and other enterprise systems.FIG. 9 is a flow diagram of an exemplary information collection process,according to one embodiment. The collection process starts (at 901) as amanager obtains API functionality and configuration capabilities from aSDN controller (at 902). The manager computes a sampling schedule as afunction of a desired performance objective and topology and sends thesampling schedule to the collector (at 903). The manager also computesand sends instruction for the collector to interact with the SDNcontroller, other enterprise systems, collect advanced statistics fromnetwork elements, and determines how to analyze, filter, and compressfrom raw data (at 904). The manager also receives raw compressed,filtered features, and other data from the collector (at 905), andindexes and stores the received raw features and data in a database interms of using time, link and other aspects such as source IP address,as keys (at 906). The manager also collects high-level policies fromuser via a user interface and other policy engines, and user feedback toaid and improve a learning algorithm (at 907).

From the set of input features and relevant input data, the presentsystem and method uses two background processes to summarize (i.e.,extract higher-layer information) and index the summarized data. Theincremental process acts upon the reception of any new raw (i.e.,un-summarized) feature data or any data update that causes previouslyindexed information to be immediately erroneous (e.g., a user changed IPaddress). This process runs a heuristic classification algorithm tosummarize the raw features. The second process is a global process thatruns periodically to update a learning model (e.g., re-training theclassification algorithm), as well as re-summarize past data. Examplesof the higher layer information include, but are not limited to:

-   -   Users;    -   Applications;    -   Protocols;    -   Device;    -   Content;    -   Network and Physical Location (Telemetry); and    -   Derived metadata, including:        -   Learned relationships between the above (e.g., User X tend            to access applications of type Y, tend to generate Z amount            of traffic),        -   Learned attributes of the above (e.g., rate of change vs.            “stickiness” of the relationships),        -   Learned behaviors about the above (e.g., this application            appears to be having TCP issues, this user appears to be            doing something malicious), and        -   Learned changes in behavior of the above (e.g., this            application has had an abnormally high set of errors, this            application is using abnormally high bandwidth).

The summarization and indexing de-duplicates data. For example, ifmultiple collectors send the same data, and the manager recognizes theduplication of data and disambiguates. In another example, if multiplecollectors see the same information from the same enterprise system, themanager recognizes the duplicate information and disambiguates.

FIG. 10 is a flow diagram of summarization and indexing processes,according to one embodiment. The summarization and indexing processstarts (at 1001) and the manager determines whether a new feature isreceived or there is a chance in network topology, statistics, and userinformation (at 1001). The manager runs incremental algorithm tosummarize and index any raw feature data, runs re-indexer to updatepreviously summarized and indexed data with changes of user or topologyinformation (at 1003). A combination of processes is used to compute ahigher-layer binding. The manager periodically (e.g., once per day) runsa global re-summarizer and re-indexer (at 1004). For example, thecollector performs a deep packet inspection (DPI) to identifyunencrypted application traffic, and the identified application is sentas a feature. Alternatively, the machine learning at the manager basedon characterizing applications by the flow or host features describedearlier can be used for encrypted traffic. User information and deviceinformation can be gleaned by accessing other enterprise systems such asactive directory, extracting key information from packets (e.g., useragent string, organizationally unique identifier (OUI)), or examiningnetwork-topology (e.g., wireless traffic comes from where the wirelessaccess points are located).

Another example concerns detecting application behaviors. For example,the machine learning at the manager can identify that the presence ofcertain packets (e.g., HTTP error packets) indicate certain types oferrors. Similarly, a heuristic algorithm that takes into account theexact physical path the traffic takes can reveal other applicationbehaviors. For example, packets are seen with increasing inter-arrivaltimes as they pass through a particular switch; this indicates acongested or misconfigured switch. An example of the outputs of theheuristic algorithm is a probabilistically ranked list of higher layerbindings.

According to one embodiment, training data is collected via user'slabelling of data. For example, a user, via a cloud portal, specifies aparticular user or application issue occurred recently. In anotherexample, when the present system and method suggests a set ofpossibilities for a given query. The user specifying which, if any, ofthose possibilities is the correct one is a useful training data.Further generalizing this, the present system and method combinesalgorithm insights from multiple networks to further enhance theclassification of the collected data.

According to another embodiment, the present system and method performs,in real time, a segment-by-segment analysis of a particularuser/application/device's traffic. To do this, the present systemcomputes the physical and logical links that the traffic of interest istaking, and alters the tapping schedule of the collector(s) so that theycollect data (e.g., packets, stats) pertaining to the physical links.Finally, the resultant features are indexed and analyzed in a similarvein to normally collected features.

Another example of summarization and indexing is computing compositemetrics from the raw features and computing and storing comparisons ofthese metrics across different dimensions. For example, the presentsystem and method computes a device quality-of-experience metric fromraw measures of response times, packet loss, etc. and compares the valueof that metric against devices of the same or different type (e.g.,iPhones), those with the same or different operating system (e.g.,Android), those connected to the same access point, etc. The computed,stored and indexed information can be quickly retrieved via a userinterface query. It can also be used for a closed loop control with aprogrammable controller. The programmable controller controls networkelements. The network manager controls the network elements via theprogrammable controller.

Cross Network Learning

The manager located in the cloud has access to systems from multipleenterprises. For example, the present system is deployed as amulti-tenant system across customers. In such a deployment, no data isshared across customers, but the processes may be shared acrosscustomers.

An example of cross network learning is to train separate classifiersfor computing higher-layer bindings from the extracted features ofseparate customer networks. The separate classifiers can be combined tocome up with an overall better classification (e.g., majority wins).Another example of cross network learning is learning the most commonqueries across networks and dedicating a higher compute power to have abetter answer for those particular queries.

Another example of cross-network learning is based on different systemdeployments that interact with each other. For example, the presentsystem is deployed at customer network 1 and customer network 2 thatsend a lot of traffic to each other. The present system and methodautomatically detects the heavy traffic, and runs a more advancedperformance testing algorithm directly between the collectors on bothcustomer networks.

Another example of cross-network learning is for predicting higher-layerperformance based on observed lower-layer characteristics of the networkand applications. For example, suppose that on one network, the presentsystem learned that high AP channel utilization results in a jitterresulting in poor real-time video application performance. The presentsystem detects the presence of high AP channel utilizations to predictpoor performance for another network that may or may not have yetdeployed a real-time video application.

Query-Ability

According to one embodiment, the present system and method providesnatural language query-ability of the network. The manager has a querybox that takes natural language type input regarding the network and itsusers/applications/devices/behaviors. Examples of natural languagequeries are:

-   -   “User X is having problem Y with application Z,”    -   “User X is experiencing slowness with salesforce.com,” and    -   “Tell me about the SAP application.”

The present system and method responds to the queries and presents aprobabilistically ranked list of answers, along with theprobabilities/confidence for each answer. The present system and methodalso presents deeper supporting evidence if requested by the user.

Summary and Example

The manager receives feature data from one or more collectors at variouslevels, for example, a flow-level, host-level, user-level, andlink-level. The manager collects and indexes the collected data in termsof flow, host, user, link, and time intervals. As a flow of feature dataarrives, the manager runs an incremental process to classify (a) anapplication that the flow corresponds to, (b) any interesting behaviorsthat the application underwent (e.g., failure to connect to a server,slow, errors), (c) a user involved in using the application, and (d) thedevices involved in using the application. Additionally, the managerties topology knowledge to an application (e.g., the location of anapplication server, network links that the application traffictraverses). This information is indexed along with each feature. Thecollector automatically runs performance tests on detected or configuredapplication servers, for example, running ping tests to the applicationservers. The performance test results are also indexed along with theapplications and features.

According to one embodiment, the present system and method provides aquery interface (e.g., web interface) to a user. The user enters aquery, for example, in a natural language form, into the user interfaceof the present system. For example, a user's query is “tell me aboutapplication X.” The present system proceeds to perform the followingsteps:

i. Query the indexed database for (a) the location of the application(e.g., on-premise, in a cloud), (b) users who were using the applicationover the last few hours, (c) the behaviors of the application, (d) thebandwidth that the application was using.

ii. Display the results of (i).

iii. Compute the links that have carried the application traffic overthe last day. Send a command to the collector to immediately collect aten-second sample of all traffic on all of the links. Send commands tothe programmable network element (e.g., via a SDN controller) to forwardthe traffic from the links to the collector.

iv. Augment the previously displayed results with those found in (iii).

Another sample query may state, “user X is having problem Y withapplication Z” (i.e., tell me about it). The manager proceeds to performthe following steps:

i. Query the indexed database for flow instances where user X was usingapplication Y. Of the behaviors recorded, rank-order the potentialproblem behaviors. Compare the corresponding features across links alongnetwork paths. Compare the features across time (i.e., historically).

ii. Display (i).

iii. Compute the links that have carried this user's application trafficover the last day. Send a command to the collector to immediatelycollect a ten-second sample of all traffic on all of these links Sendcommands to the programmable network element (e.g., via a SDNcontroller) to forward the traffic from those links to the collector.

iv. Augment the previously displayed results with those found in (iii).

Control

According to some embodiments, the present system and method involvesusing the visibility of the network and controlling the network. Anexample of controlling the network is enforcing a higher-layer policythroughout the network. Another example is automatic problem andsecurity/anomaly/performance remediation where applicable. The presentsystem and method may implement a network control in (a) a manual, orprescribed control, and (b) an automatic closed loop control. In bothcases, one of the distinctions from the visibility perspective is thatthe binding of a higher layer policy or a control objective needs to betracked to the specific low layer control primitives that the underlyingnetwork elements can be programmed with. Examples of the high-levelcontrol objectives include, but are not limited to:

-   -   Block user X from accessing the network,    -   Maintain high performance for Application Y,    -   Detect and mitigate denial of service (DOS) attacks, and    -   Prioritize user class Z traffic.

For a manual/prescribed control, the control instructions that achieve ahigh level objective are computed and presented to the user, but notautomatically programmed into the network elements. In addition,specific network elements that require a new or updated configurationbased on the control instructions are computed as a function of networktopology and presented to the user. The present system computes how thecontrol is to be achieved in a distributed manner. The controlinstruction sets may be probabilistically ranked in the order ofpredicted effectiveness. While an explicit machine-to-machineprogrammability (e.g., SDN controller) may not be required in someembodiments, it may be required for the present system to discover theconfiguration state and capabilities of the various network elements inother embodiments. The present system takes into account specific lowlevel control primitives that the network elements can be configuredwith. For example, many network elements have IP, MAC, and TCAM hardwaretables of different sizes that are programmable with differentprimitives.

According to some embodiments, the present system and method dynamicallytracks the bindings between user and (IP address, MAC address, physicalport) as a user changes devices, plugs into a different sub-network, andreceives a new IP address from a dynamic host configuration protocol(DHCP) server. According to some embodiments, the present system andmethod binds an application/network performance issue to specifictraffic forwarding decisions (e.g., application slowness is caused by aset of particular source/destination IP address pairs that are highlyutilizing a particular link) or a network configuration (e.g., amisconfigured maximum transmission unit (MTU)). According to someembodiments, the present system and method ties a particular anomaloustraffic behavior to a specific user/application/device, and further toparticular IP/MAC addresses.

According to some embodiments, the present system and method takes intoaccount the topology and capabilities of the underlying networkhardware. For example, if one is trying to use a pure layer 2 switch toenforce a user policy, it would be required to dynamically track theUser MAC address binding, and use only MAC addresses for programmingrules into the switch. An example of taking the topology into account,the present system and method tries to enforce a policy as close to theedge of the network as possible, which current firewalls, usuallydeployed inline at logical or physical network choke points, cannot do.The rules programmed to the network elements can be changed in a closedloop manner when the higher-layer to lower-layer bindings change.

FIG. 11 is a flow diagram of a control loop, according to oneembodiment. The control loop starts (at 1101), and the managerdetermines whether there are unsatisfied high-level control objective(at 1102). The manager branches off based on a control method (at 1103).For a manual control method, the manager computes the optimizedlower-level rules and topologies to send to the network controller baseon, but not limited to, 1) the high-level control objective, 2)estimated higher layer bindings values and associated uncertainties, 3)configuration capability and current configuration of underlying networkelements, and 4) other information such as network topology, statistics,tolerable configuration changes (at 1104). The manager presents thecontrol method of specific network elements to achieve the high-levelcontrol objective to the user (at 1105). For an automatic control, themanager computes the initial update control to the programmable networkelements based on, but not limited to, 1) the high-level policies,problems, security requirements, anomalies, 2) estimated higher layerparameter values and associated uncertainties, 3) configurationcapability and current configuration of underlying network elements, 4)other information such as network topology, statistics, tolerableconfiguration change, 5) measurement of effectiveness of the controlpolicy, and 6) control loop parameters such as stability, oscillation,timescale (at 1106). The manager sends the control policy parameters tothe programmable network elements (at 1107), and observes the networkand measures effectiveness of the control policy with respect to thehigh-level policy (at 1108).

As an example of manual/prescribed control, the present system andmethod enforces a high level objective of blocking user X from thenetwork. To do this, the present system and method first derives the IPaddresses that user X corresponds to. Then, the present system andmethod computes a logical choke point to apply the policy effectively.For example, the logical choke point corresponds to the routers on thesubnets of user X's IP address. The output of the present systemincludes a set of commands at each of the routers that results in thetraffic from/to those IP addresses being dropped. An alternative outputis a set of commands to a SDN controller to implement a desired control.

For an automatic control, the present system and method programs thenetwork elements in a closed loop manner to achieve and maintain ahigh-level control objective. The automatic control is based on aninherent assumption that the underlying network has programmable (e.g.,SDN-enabled) network elements. In addition to the binding ofhigher-layer objectives to low-layer programmable primitives and takinginto account the configuration state and capabilities of the underlyingnetwork elements, the present system and method computes a dynamiccontrol loop. The present system and method first applies a possiblecontrol (e.g., a gain) and checks to see if a high-level objective isachieved. If so, the present system and method backs off the remediationand/or applies a different but lighter remediation and checks again tosee if the high-level objective is still achieved. If not, the presentsystem and method attempts to apply a heavier control and/or re-diagnosethe higher-layer objective to low layer control primitives binding andapply a different control. This procedure is also depicted in FIG. 11.The first step of the closed loop control may be different from thesteps provided by the manual control. Additionally, factors such asstability, oscillation and timescale of response may be taken intoaccount in the setup of the control loop.

The automatic closed loop control can be applied to the example ofblocking user X from the network. In this example, the present systemand method programs rules to drop traffic from/to user X's IPaddress(es) at the routers in the network. Assuming that works, thepresent system and method tries to program only user X's default gatewayrouter with a rule. If it fails, the present system and method appliesmore rules to other routers as well as and/or blocks certain ports andcontinues. When the user X comes in on new IP address(es), the presentsystem and method automatically adjusts to the changed network topology.

Another use case of an automatic closed loop control is where thecontrol objective is to maintain high performance for application X. Inthis case, the present system and method simply programs rules thatplace all traffic corresponding to that application into the highestperforming queue. If improved application X performance is not observer,the present system and method attempts to program rules that re-routesor rate-limits traffic from applications that share common network linkswith application X. If improvements are observed, the present system andmethod restores the performance of other applications.

An example of a higher layer policy (for manual or automatic control) is“Prioritize traffic from employees using business applications such asSalesforce.com or Workday, over casual traffic such as traffic fromguest users using a different set of applications.” To implement thishigher layer policy, the present system and method dynamically tracksthe session 5-tuples for these combinations, and computes a minimal setof rules necessary for the enforcement, and dynamically tracks andprograms.

According to some embodiments, the present system and methodautomatically provides remedies to network problems. For example, a userenters in a query of the form “user X is having problem Y withapplication Z,” and the present system and method provides thetop-ranked answer (i.e., the answer with confidence greater than acertain threshold) that “there is congestion on common network linkscaused by users using application W.” If automatic remediation isenabled for this particular query, the manager sends instructions to thecollector to command the SDN controller to tell the appropriate networkelements to (a) prioritize user X application Z traffic over othertraffic, or (b) disallow traffic involving application W. The (b)remediation approach may require additional policy permission from theoperator due to the restrictive nature of the traffic disallowingpolicy.

Referring to FIG. 4 as an example of the remediation process, supposethat user X is “attached” to switch s3 and that application Z server is“attached” to switch s4. The policy to prioritize user X application Ztraffic may be applied by the SDN controller that sends rules to switchs3 that matches user X's IP address (as source IP) and the applicationserver IP address (as destination IP), and has an action that marks theIP diffsery code point (DSCP) bits to represent the highest class ofservice. Similarly, the reverse rule is applied to switch s4 (i.e., withthe source and destination IP addresses flipped).

Alternatively, the rules may be applied to all switches along thecommunication path. These rules have similar match fields, but theaction field directly sends the traffic to the highest priority queue.If the policy is to drop user X application Z traffic, the rules areapplied to the edge switches s3 and s4, respectively. This is ausefulness technique since the rules do not need to be appliedeverywhere in the network.

Another example of the automated remediation process is in theconfiguration domain. For example, for a query “there is a problem withapplication X,” suppose that the top-ranked answer is “the problemappears to be that switch Y is dropping packets due to a misconfiguredmaximum transmission unit (MTU) value.” The present system and methodremediates this situation automatically by sending instructions to thecollector to command the SDN controller to reconfigure the MTU value ofthe appropriate switch.

According to some embodiments, one of the applications of turningvisibility into control is a full-fledged distributed firewall. Forexample, the operator sets up a policy “user X cannot access applicationY,” or “user X may be barred from the network for Y minutes after Zfailed logon attempts.” In other example, the operator sets up a policyto isolate (e.g., on a quarantine VLAN®) a user whose traffic exhibitsmalicious or anomalous behavior. The detection and manual or automaticremediation of an anomaly (e.g., a detected DOS attack) can also beaddressed within the control framework of the present system and method.

FIG. 12 illustrates an exemplary computer architecture that may be usedfor the present system, according to one embodiment. The exemplarycomputer architecture may be used for implementing one or morecomponents described in the present disclosure including, but notlimited to, the present system. One embodiment of architecture 1200includes a system bus 1201 for communicating information, and aprocessor 1202 coupled to bus 1001 for processing information.Architecture 1200 further includes a random access memory (RAM) or otherdynamic storage device 1203 (referred to herein as main memory), coupledto bus 1201 for storing information and instructions to be executed byprocessor 1202. Main memory 1203 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions by processor 1202. Architecture 1200 may also include aread only memory (ROM) and/or other static storage device 1204 coupledto bus 1201 for storing static information and instructions used byprocessor 1202.

A data storage device 1205 such as a magnetic disk or optical disc andits corresponding drive may also be coupled to architecture 1200 forstoring information and instructions. Architecture 1200 can also becoupled to a second I/O bus 1206 via an I/O interface 1207. A pluralityof I/O devices may be coupled to I/O bus 1206, including a displaydevice 1208, an input device (e.g., an alphanumeric input device 1209and/or a cursor control device 1210).

The communication device 1211 allows for access to other computers(e.g., servers or clients) via a network. The communication device 1211may include one or more modems, network interface cards, wirelessnetwork interfaces or other interface devices, such as those used forcoupling to Ethernet, token ring, or other types of networks.

The foregoing description, for purposes of explanation, uses specificnomenclature and formula to provide a thorough understanding of thedisclosed embodiments. It should be apparent to those of skill in theart that the specific details are not required in order to practice theinvention. The embodiments have been chosen and described to bestexplain the principles of the disclosed embodiments and its practicalapplication, thereby enabling others of skill in the art to utilize thedisclosed embodiments, and various embodiments with variousmodifications as are suited to the particular use contemplated. Thus,the foregoing disclosure is not intended to be exhaustive or to limitthe invention to the precise forms disclosed, and those of skill in theart recognize that many modifications and variations are possible inview of the above teachings.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a disclosed embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed:
 1. A system for simultaneously and centrally analyzinga plurality of networks, the system comprising: one or more collectorsconfigured to receive network traffic data from a plurality of networkelements in the plurality of networks, wherein a first network of theplurality of networks is from a first company and a second network ofthe plurality of networks is from a second company; a non-transitorycomputer readable manager memory; and a remote network managercomprising a network interface and configured to connect over theInternet to the one or more collectors via the network interface, theremote network manager further configured to simultaneously andcentrally analyze (1) the network traffic data from the plurality ofnetwork elements in the plurality of networks and (2) network managementdata from a plurality of enterprise systems in the plurality ofnetworks, wherein the network management data includes L1 through L7network topology data, network configuration data, and simple networkmanagement protocol data; wherein the remote network manager is storedin the non-transitory computer readable manager memory that is executedby a manager processor, wherein the remote network manager combines thenetwork traffic data from the plurality of networks and the networkmanagement data from the plurality of enterprise systems in theplurality of networks into combined cross-network data from multiplecompanies, simultaneously and centrally analyzes the combinedcross-network data from the multiple companies within the plurality ofnetwork systems, learns a network pattern from the first network of thefirst company within the plurality of networks, and predicts a networkbehavior of the second network of the second company within theplurality of networks, wherein the remote network manager identifies anetwork control objective for the network, identifies a programmableparameter of a programmable network element to achieve the networkcontrol objective, and programs the programmable network element that isa programmable switch, router, or wireless access point, wherein thenetwork control objective is security or performance remediation,wherein the remote network manager identifies specific applications ordevices causing security or performance issues, and wherein theprogrammable parameter identified by the remote network manager isblocking the specific applications or devices causing the security orperformance issues, wherein the remote network manager computes aninitial control policy to the programmable network elements usinghigh-level policies, security requirements, and any anomalous trafficbehavior; configuration capability and current configuration ofunderlying network elements; network topology, statistics, and tolerableconfiguration change; and control loop parameters including stability,oscillation, and timescale, and wherein the remote network manager sendscontrol policy parameters to the programmable network elements, observesthe network, measures effectiveness of the initial control policy withrespect to the high-level policy, and computes an updated control policyto send to the programmable network elements.
 2. The system of claim 1,wherein the remote network manager is configured to predict performanceof an application, a user, or a device based on observed characteristicsof the network using network protocol level metrics and metadata.
 3. Thesystem of claim 1, wherein the network pattern is a protocol levelmetric of the network and metadata of network traffic data.
 4. Thesystem of claim 1, wherein the remote network manager controls aplurality of network elements.
 5. The system of claim 1, furthercomprising one or more collectors that are disposed in a network systemof the plurality of network systems and configured to receive networktraffic data from a plurality of network elements in the network system.6. The system of claim 5, wherein the remote network manager or the oneor more collectors are further configured to index the network enablingefficient search and retrieval of the metadata and learned networkpatterns.
 7. The system of claim 5, wherein the one or more collectorsare further configured to receive statistics about the network, topologyinformation about the network, input from one or more enterprisesystems, or combinations thereof.
 8. The system of claim 5, furthercomprising a programmable network element disposed in the networksystem, wherein the one or more collectors program the programmablenetwork element to configure the programmable network element to sendfiltered network traffic data to the one or more collectors.
 9. Thesystem of claim 5, further comprising a programmable network elementdisposed in the network system, wherein the remote network managerprograms the programmable network element to configure the programmablenetwork element to send filtered network traffic data to the one or morecollectors.
 10. The system of claim 9, wherein the remote networkmanager applies a control loop to determine whether a network controlobjective is met after programming the programmable network element. 11.The system of claim 9, wherein the remote network manager affects anetwork policy by programming a programmable network element with acontrol primitive.
 12. The system of claim 11, wherein the controlprimitive includes an access control list (ACL), quality of service(QoS), rate limit settings, or combinations thereof.
 13. The system ofclaim 11, wherein the remote network manager maintains a relationshipbetween a network policy and the control primitive in a database. 14.The system of claim 1, wherein network analysis is performed usingmetadata.
 15. The system of claim 1, wherein the remote network manageris disposed in a cloud and connected to the one or more collectors overthe Internet.
 16. The system of claim 1, wherein metadata istime-aligned with data received from one or more enterprise systems. 17.The system of claim 1, wherein the one or more collectors areprogrammable, and wherein the remote network manager programs the one ormore collectors to collect different types of metadata.
 18. The systemof claim 1, wherein the remote network manager de-duplicates themetadata received from the one or more collectors.
 19. The system ofclaim 18, wherein the higher layer information includes a relationshipor binding of higher layer data to lower layer data, wherein higherlayer data corresponds to users, applications, devices, or combinationsthereof, and wherein lower layer data corresponds to IP and MACaddresses, ports, or combinations thereof.
 20. The system of claim 1,wherein the system extracts features from the network traffic data,summarizes data regarding extracted higher layer information from thenetwork traffic data, and indexes the summarized data in a database forpattern identification.
 21. The system of claim 1, wherein the systemlearns a pattern by identifying specific network, protocol, and wirelessmetrics to determine application performance.
 22. The system of claim 1,wherein the remote network manager calculates a quality of experience ofa user, an application, or a device, based on the metadata received fromthe one or more collectors.
 23. A system for simultaneously andcentrally analyzing a plurality of networks, the system comprising: oneor more collectors configured to receive network traffic data from aplurality of network elements in the plurality of networks, wherein afirst network of the plurality of networks is from a first company and asecond network of the plurality of networks is from a second company; anon-transitory computer readable manager memory; and a remote networkmanager comprising a network interface and configured to connect overthe Internet to the one or more collectors via the network interface,the remote network manager further configured to simultaneously andcentrally analyze (1) the network traffic data from the plurality ofnetwork elements in the plurality of networks and (2) network managementdata from a plurality of enterprise systems in the plurality ofnetworks, wherein the network management data includes L1 through L7network topology data, network configuration data, and simple networkmanagement protocol data, wherein the remote network manager is storedin the non-transitory computer readable manager memory that is executedby a manager processor, wherein the remote network manager combines thenetwork traffic data from the plurality of networks and the networkmanagement data from the plurality of enterprise systems in theplurality of networks into combined cross-network data from multiplecompanies, simultaneously and centrally analyzes the combinedcross-network data from the multiple companies within the plurality ofnetwork systems, learns a network pattern from the first network of thefirst company within the plurality of networks, and applies a networkpolicy to the second network of the second company within the pluralityof networks, wherein the remote network manager identifies specificapplications or devices causing security or performance issues, andwherein the remote network manager blocks the specific applications ordevices causing the security or performance issues, wherein the remotenetwork manager computes an initial control policy to the programmablenetwork elements using high-level policies, security requirements, andany anomalous traffic behavior; configuration capability and currentconfiguration of underlying network elements; network topology,statistics, and tolerable configuration change; and control loopparameters including stability, oscillation, and timescale.
 24. Thesystem of claim 23, wherein the network policy is controlling one ormore of applications, users, or devices of the network.
 25. A method forsimultaneously and centrally analyzing a plurality of networks, themethod comprising: providing one or more collectors configured toreceive network traffic data from a plurality of network elements in theplurality of networks, wherein a first network of the plurality ofnetworks is from a first company and a second network of the pluralityof networks is from a second company; providing a non-transitorycomputer readable manager memory; providing a remote network managercomprising a network interface and configured to connect over theInternet to the one or more collectors via the network interface, theremote network manager further configured to simultaneously andcentrally analyze (1) the network traffic data from the plurality ofnetwork elements in the plurality of networks and (2) network managementdata from a plurality of enterprise systems in the plurality ofnetworks, wherein the network management data includes L1 through L7network topology data, network configuration data, and simple networkmanagement protocol data, wherein the remote network manager is storedin the non-transitory computer readable manager memory that is executedby a manager processor; sending the metadata to the remote networkmanager, wherein the remote network manager combines the network trafficdata from the plurality of networks and the network management data fromthe plurality of enterprise systems in the plurality of networks intocombined cross-network data from multiple companies; simultaneously andcentrally analyzing the combined cross-network data from the multiplecompanies within the plurality of networks; learning a network patternfrom the first network of the first company within the plurality ofnetworks; and applying a network policy from the second network of thesecond company within the plurality of networks, wherein the remotenetwork manager identifies applications or devices causing security orperformance issues, and wherein the remote network manager blocks thespecific applications or devices causing the security or performanceissues, wherein the remote network manager sends control policyparameters to the programmable network elements, observes the network,measures effectiveness of the initial control policy with respect to thehigh-level policy, and computes an updated control policy to send to theprogrammable network elements.